(19) 



J 



Europe isches Patentamt 
European Pat nt Office 
Office uropeen des brevets 



EP 1 318 668 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(43) 


Date of publication: 


(51) Intel. 7 : H04N 5/235 




11.06.2003 Bulletin 2003/24 


(21) 


Application number: 02026605.2 




(22) 


Date of filing: 28.11.2002 




(84) 


Designated Contracting States: 


(72) Inventors: 




AT BE BG CH CY CZ DE DK EE ES Fl FR GB GR 


• Barkan, Stanley 




IE IT LI LU MC NL PT SE SK TR 


30825 D.N. Hof HaCarmel (IL) 




Designated Extension States: 


• Carmi, Han 




AL LT LV MK RO SI 


53301 Givatayim (IL) 


(30) 


Priority: 05.12.2001 US 335796 P 


(74) Representative: Hofmann, Harald 




08.10.2002 US 265597 


Sonnenberg Fortmann, 






Patent- und Rechtsanwalte, 


(71) 


Applicant: Creo IL. Ltd. 


Herzogspitalstrasse 10a 




46103 Herzlia Pituah (IL) 


80331 Munchen (DE) 



(54) A system and method for the formation of multiple exposure images 



(57) An image capture apparatus creates multiple 
exposure images by combining raw sensor output data 
prior to other image processing. The apparatus consists 
of a sensor for capturing current raw image data and an 
image combiner. The image combiner forms a multiple 
exposure image by linearly combining stored raw image 
data with current raw image data from the sensor. 
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D scription 

[0001 J The present invention relates to digital photog- 
raphy and more particularly but not exclusively to mul- 
tiple exposure digital photography. 5 
[0002] In state-of-the-art film cameras, it is possible 
to expose a single frame of film several times in succes- 
sion by preventing the film from advancing by mechan- 
ical means. This multiple exposure option enables the 
photographer to solve a number of difficulties encoun- 
tered when the lighting is not optimal, besides being able 
to create special effects. Creating a multiple exposure 
photograph with a digital camera by repeatedly expos- 
ing the sensor to light has been problematic, due to 
noise buildup on the sensor. Digital cameras therefore 
do not currently offer this option. The photographer is 
expected to expose a sequence of images and then 
combine them using image-processing software such 
as Photoshop, available from Adobe Systems Incorpo- 
rated, of San Jose, USA. An example of this combining 
technique is provided in an article at the following Inter- 
net website: 

http://www.dpreview.com/leam/lmage_Techniques/ 
Double_Exposures_01.htm included herein as appen- 
dix A. 

[0003] There are numerous cases where the combi- 
nation of images is a simple sum. In these cases the use 
of post-capture software is not optimal. Prior to the im- 
age combination process the images are rendered to be 
color correct for viewing. The pixel values are then not 
linearly proportional to the original light. 
[0004] Suppose an image has been rendered, yield- 
ing the following non-linear transformation of the pixel 
intensity levels. 
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Doubling output levels of (0, 10, 20) after rendering 
yields the result (0,50,80). However if the uncorrected 
input values are first doubled and then transformed, the 
original levels of (0,10,20) yield the result (0,40,100). 
Since the goal is to add the input values (equivalent to 
a longer exposure time), it is clear that the non-linear 
transformation has introduced a distortion of the inten- 
sity levels. With a limited number of bits, reversing the 
non-linear transformation of the data prior to addition 
causes a loss in accuracy. 

[0005] One typical use of the multiple exposure tech- 
nique is when images are combined to achieve depth of 
field. In this case, the photographer wishes to keep the 
aperture at minimal size and the flash equipment is not 



powerful enough to provide correct exposure. It is pos- 
sible in this case to increase the exposure time and to 
program the flash to fire several times. However, this 
requires expensive equipment. Also, during the flash re- 
covery time, noise may build up while no information is 
added. Another example where it is critical to add light 
in linear fashion is when the scene is lit simultaneously 
from indoors and from outdoors, for example when there 
is a window in the image. In this case it is customary to 
shoot several images by repeated exposure of a single 
frame: with the window open, with the window closed, 
and with and without flash. In the analog case, all the 
images are thus linearly added on to the single multiply 
exposed frame. However, in the digital case, combining 
such differently lit images using image-processing soft- 
ware after they have been captured digitally and ren- 
dered to a viewing space, cannot be done in a linear 
fashion for the reasons mentioned above. The transfor- 
mations performed on the image are nonlinear and irre- 
versible. Likewise, adding the images directly on the 
sensor is not possible due to noise considerations, as 
mentioned above. 

[0006] Image combination has been proposed to al- 
leviate the problem of the limited dynamic range of dig- 
ital camera image sensors. In high contrast scenes, the 
dynamic range of the camera sensor is often not ade- 
quate to provide detail for both dark and light portions 
of the image, even using a charge-coupled device 
(CCD) sensor. InU.S. Pat. 6,177.958 to Anderson, after 
a high contrast scene is detected, the image is captured 
twice, at different exposures. The bright image and the 
dark image are then combined to increase image dy- 
namic range. A number of methods for combining the 
two images are described. These methods include: (1) 
determining an offset to achieve spatial alignment and 
aligning the images on a pixel-by-pixel basis, (2) deter- 
mining common areas of the two images and adjusting 
the exposure overlapping areas so that the common ar- 
eas are equal in brightness, (3) selecting pixels form the 
dark image where the pixels are below the darkest area 
of the exposure overlapped area, or (4) selecting pixels 
from the light image where the pixel is above the bright- 
est area of the exposure overlapped area. In all cases 
the bright and dark images are combined non-linearly, 
under the assumption that the images being combined 
are of essentially the same scene. . Similar solutions are 
presented in U.S. Patents 6,011,251 to Dierickx, et al., 
5,247,366 to Ginosar, et al., 5,1 44,442 to Hilsenrath, et 
al., and 4,647,975 to Alston, et al. Additional solutions 
are found in World IPO 0,113,171 to Crawford, et al., 
and European Patents 0,982,983 to Inagaki, et al., and 
0,910,209 to Yoneyama, et al. These patents disclose 
various methods of increasing image dynamic range, by 
combining images of a scene, generally taken at signif- 
icantly different exposures, in such a way as to enhance 
the darkest and brightest components of the image. 
Documents referred to above are hereby incorporated 
by reference. 
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[0007] Current analog cameras, which do offer multi- 
ple exposure features, do not compute the correct 
number of frames to be taken to arrive at the correct 
exposure. The photographer must measure the light 
that is obtained from a single exposure and compute the 5 
number of frames required for correct exposure at the 
desired aperture setting. If the flash strength is not con- 
sistent, this number may not be exactly predictable. Fur- 
thermore, camera settings, such as aperture or expo- 
sure time, are not always controllable when the camera 
is in multiple exposure mode. 

[0008] It is a first object of the present embodiments 
to provide a multiple exposure feature for digital camer- 
as, which combines multiple images linearly. 
[0009] It is a further object of the present embodi- 
ments to provide a digital camera in which the multiple 
exposure process can be controlled, either manually or 
automatically, so as to easily obtain the combination of 
images required for correct exposure. 
[0010] More generally, it is an object of the present 
embodiments to provide a camera that is able to monitor 
actual accumulated light over a series of exposures, so 
as to be able to terminate the multiple exposure process 
when optimal overall light levels have been obtained. 
[001 1 ] According to a first aspect of the present inven- 
tion there is thus provided an image capture apparatus 
for the formation of a multiple exposure image, compris- 
ing: 

an image capture sensor for capturing images, and 
a linear image combiner, associated with said im- 
age capture sensor, for linearly combining raw im- 
age capture data from said sensor, thereby to form 
said multiple exposure image. 

[001 2] The apparatus may comprise a buffer memory 
at said sensor output for storing said raw image data, 
said buffer memory being operatively associated with 
said linear image combiner to allow said image combin- 
er access to said raw image data. 
[0013] Preferably, said linear image combiner is fur- 
ther operable to store said multiple exposure image in 
said buffer memory. 

[0014] Preferably, said apparatus comprises a digital 
camera. 

[0015] Preferably, said linear image combiner is op- 
erable to form said multiple exposure image as a linear 
combination of stored raw image data and a current raw 
image data capture. 

[0016] Preferably, said linear combination is one of a 
group comprising: a linear fraction of a sum of said 
stored raw image data and said current raw image data 
capture, an average of said stored raw image data and 
said current raw image data capture, and a sum of said 
stored raw image data and a linear fraction of said cur- 
rent raw image data capture. 

[0017] Preferably, said linear image combiner is con- 
trollable by a user. 



[0018] Preferably, said linear image combiner is op- 
erable to combine said raw image data automatically, 
so as to satisfy a predetermined exposure criterion for 
said multiple exposure image. 

[0019] Preferably, said criterion comprises having a 
pixel intensity level within a segment of said multiple ex- 
posure image exceed a predetermined threshold. Pref- 
erably, said linear image combiner is operable to form 
said multiple exposure image from more than one image 
stored in said buffer memory. 

[0020] The apparatus may comprise a controller, for 
controlling apparatus functions. Preferably, said control- 
ler is operable to control said functions in accordance 
with information obtained from said image combiner. 
[0021 ] Preferably, one of said functions comprises ac- 
tivating and deactivating a multiple exposure mode. 
[0022] Preferably, one of said functions comprises se- 
lecting a segment of an image. 

[0023] Preferably, one of said functions comprises se- 
lecting and deselecting an automatic multiple exposure 
mode. 

[0024] Preferably, one of said functions comprises 
capturing an image. 

[0025] Preferably, said apparatus comprises a digital 
camera and one of said functions comprises controlling 
camera settings. 

[0026] Preferably, said settings comprise camera ap- 
erture, camera speed, and flash operation. 
[0027] The apparatus may comprise a converter, for 
converting current raw image data into a digitized im- 
age. 

[0028] Preferably, said buffer memory comprises a 
random access memory (RAM). The apparatus may 
comprise a display. 

[0029] Preferably, said display is operable to display 
a selected image. 

[0030] Preferably, said selected image is one of a 
group comprising: a current image, said multiple expo- 
sure image, and an image stored in said buffer memory. 
Preferably, said display is operable to display a histo- 
gram for a selected image. Preferably, said histogram 
indicates the relative distribution of light intensities for 
said selected image. 

[0031] Preferably, said histogram indicates a current 
state of exposure of said selected image. 
[0032] Preferably, said selected image is one of a 
group comprising: a current image, said multiple expo- 
sure image, and an image from buffer memory. 
[0033] The apparatus may comprise a pointer, for se- 
lecting a segment of a displayed image. 
[0034] Preferably, said sensor comprises a charge- 
coupled device (CCD) sensor. Preferably, said sensor 
comprises a complementary metal oxide semiconductor 
(CMOS) sensor. 

[0035] The apparatus may comprise an image proc- 
essor for processing raw image data. 
[0036] Preferably, said processing comprises render- 
ing said raw image data into a viewable image. 
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[0037] The apparatus may comprise a main memory 
for storing said processed image data. 
[0038] According to a second aspect of the present 
invention there is preferably provided a method for cre- 
ating multiple exposure images by linearly combining 
raw capture data from a sensor, comprising: 

obtaining current raw image data; 

storing said current raw image data in memory; and 

performing a multiple exposure iteration step by: 

obtaining new current raw image data; 
linearly combining stored raw image data with 
said current raw image data to form a multiple 
exposure image; and 

storing said multiple exposure image in said 
memory. 

[0039] Preferably, said multiple exposure iteration 
step further comprises the steps of: 

determining if said multiple exposure image is cor- 
rectly exposed; 

if said multiple exposure image is correctly ex- 
posed, discontinuing said multiple exposure proc- 
ess; and 

if said multiple exposure image is not correctly ex- 
posed, performing a further multiple exposure iter- 
ation step. 

[0040] Preferably, said stored raw image data com- 
prises more than one image stored in said memory. 
[0041] Preferably, said method is performed by a dig- 
ital camera. 

[0042] The method may comprise adjusting camera 
settings. 

[0043] Preferably, said multiple exposure iteration 
step further comprises adjusting camera settings. 
[0044] The method may comprise adjusting said cam- 
era settings automatically. Preferably, said multiple ex- 
posure iteration step further comprises adjusting said 
camera settings automatically. 

[0045] Preferably, said multiple exposure iteration 
step further comprises displaying a selected image. 
[0046] Preferably, said selected image is one of a 
group comprising: a current image, said multiple expo- 
sure image, and an image stored in said memory. 
[0047] Preferably, said multiple exposure iteration 
step further comprises selecting at least one segment 
of said displayed image. 

[0048] The method may comprise analyzing said se- 
lected segments to determine image exposure. 
[0049] Preferably, said multiple exposure iteration 
step further comprises displaying a histogram of a se- 
lected image. 

[0050] Preferably, said selected image is one of a 
group comprising: a current image, said multiple expo- 
sure image, and an image stored in said memory. 
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[0051] Preferably, analyzing said histogram to deter- 
mine image exposure. 

[0052] Preferably, said histogram indicates the rela- 
tive distribution of light intensities for said selected im- 
5 age. 

[0053] Preferably, said histogram indicates a current 
state of exposure of said selected image. 
[0054] The method may comprise selecting a manner 
of combining said images. Preferably, said images are 
10 combined automatically according to said selected 
manner. 

[0055] The method may comprise selecting a criterion 
for determining if said multiple exposure image is cor- 
rectly exposed. 

is [0056] Preferably, said criterion comprises having a 
pixel intensity level within a segment of said multiple ex- 
posure image exceed a predetermined threshold. Pref- 
erably, said memory comprises a RAM. 
[0057] The method may comprise digitizing said cur- 

20 rent raw image data. 

[0058] According to a third aspect of the present in- 
vention there is provided an image capture apparatus 
comprising a sensor for capturing raw image data, an 
image buffer for storing said captured raw image data, 

25 an image processor for processing said captured data 
into a displayable image file and a memory for storing 
said image file, the apparatus further comprising an im- 
age combiner associated with said image buffer for per- 
forming linear combinations between different captures 

30 of said raw image data, therefrom to form multiple ex- 
posure images. 

[0059] Preferably, said image combiner is operable to 
continue to accumulate combinations using succeeding 
image captures until a predetermined light level is 

35 reached within said combination. 

[0060] For a better understanding of the invention and 
to show how the same may be carried into effect, refer- 
ence will now be made, purely by way of example, to 
the accompanying drawings. 

40 [0061 ] With specific reference now to the drawings in 
detail, it is stressed that the particulars shown are by 
way of example and for purposes of illustrative discus- 
sion of the preferred embodiments of the present inven- 
tion only, and are presented in the cause of providing 

45 what is believed to be the most useful and readily un- 
derstood description of the principles and conceptual 
aspects of the invention. In this regard, no attempt is 
made to show structural details of the invention in more 
detail than is necessary for a fundamental understand- 

50 ing of the invention, the description taken with the draw- 
ings making apparent to those skilled in the art how the 
several forms of the invention may be embodied in prac- 
tice. In the accompanying drawings: 

55 Figure 1 shows a simplified block diagram of an im- 
age capture apparatus for the formation of a multi- 
ple exposure image according to a preferred em- 
bodiment of the present invention. 
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Figure 2 shows a simplified block diagram of an im- 
age capture apparatus for the formation of a multi- 
ple exposure image according to a further preferred 
embodiment of the present invention. 
Figure 3 is a simplified flowchart of a method for ere- 5 
ating multiple exposure images. 
Figure 4 is a simplified flowchart of a manual oper- 
ation mode for creating multiple exposure images. 
Figure 5 is a simplified flowchart of an automatic op- 
eration mode for creating multiple exposure imag- 
es. 

[0062] Multiple exposure is a well-known photogra- 
phy technique. In film cameras the technique is 
achieved by photographing images without advancing 
the film between photographs. The result is a linear sum 
of the photographed images. In digital photography the 
effect is obtained by using image processing software 
to display and combine previously photographed imag- 
es, which have been rendered for viewing. Because the 
processing is carried out on the rendered image, the pix- 
el brightness levels are no longer proportional to the in- 
put light. Thus combining at the post-rendering stage in- 
troduces non-linear effects into the combined image. In 
the camera itself, the sensor captures an undistorted im- 
age, the undistorted or raw image is flushed to a raw 
image buffer, and from the image buffer the raw image 
is rendered for viewing on a screen and formatted for 
storage as an image file. Combining the photographed 
images after storage ordinarily would not help because 
the stored image file is rendered. 
[0063] One possible technique for creating multiple 
exposures in a digital camera is to re-expose the camera 
sensor to light without flushing the sensor. Re-exposing 
the sensor creates a linearly summed image as desired. 
However, unlike in a film camera, the resultant digital 
image is degraded by accumulated noise on the sensor. 
Additionally, the process is relatively inflexible. For ex- 
ample, the resultant image cannot be formed as an av- 
erage of the photographed images. 
[0064] However, the problem may be overcome by re- 
taining the raw image in the image buffer and adding 
subsequent output from the sensor directly to the raw 
image in the image buffer. 

[0065] Before explaining at least one embodiment of 
the invention in detail, it is to be understood that the in- 
vention is not limited in its application to the details of 
construction and the arrangement of the components 
set forth in the following description or illustrated in the 
drawings. The invention is applicable to other embodi- 
ments or of being practiced or carried out in various 
ways. Also, it is to be understood that the phraseology 
and terminology employed herein is for the purpose of 
description and should not be regarded as limiting. 
[0066] Reference is now made to Figure 1, which 
shows a simplified block diagram of an image capture 
apparatus 100, which creates multiple exposure images 
from raw capture data from a sensor. The image capture 



apparatus 1 00 consists of an image capture sensor 1 1 0 
and linear image combiner 120. Light impinging on sen- 
sor 110 is captured to form current raw image data. In 
the following description, the term raw image data refers 
to image data as flushed from the sensor and which has 
not undergone significant nonlinear processing, for ex- 
ample for storage, viewing, or like purposes. The pos- 
sibility of having carried out digitization or equalization 
procedures is not excluded. Linear image combiner 1 20 
combines the raw image data from the sensor linearly, 
to form a multiple exposure image. In the preferred em- 
bodiment apparatus 1 00 is a digital camera, which gen- 
erally contains a CCD or CMOS sensor. 
[0067] The image capture apparatus 100 enables a 
user to start and stop accumulating image data with 
complete freedom. During the time the accumulation is 
stopped, the user can make any desired changes to the 
scene being captured, or to camera settings; exposure 
settings can be changed, a flash unit can be adjusted, 
and so forth. 

[0068] In a first preferred embodiment the image cap- 
ture apparatus 100 also contains a buffer memory 130 
(also referred to as image buffer). Buffer memory 130 
stores image data as it is flushed from the sensor 110. 
Additionally, buffer memory 130 may store combined im- 
age data obtained from linear image combiner 120. All 
or a portion of the image data stored in buffer memory 
1 30 is available for processing by the linear image com- 
biner 1 20. Buffer memory 1 30 may be a random access 
memory (RAM). Linear image combiner 1 20 creates the 
multiple exposure image by linearly combining image 
data from the buffer memory 1 30 with current image da- 
ta obtained from the sensor 11 0, or with other image da- 
ta from the buffer memory 130. When the desired mul- 
tiple exposure image is achieved, the image may be 
transferred from the image buffer to another section of 
the buffer memory 130, or rendered into a viewable im- 
age and stored in main memory 1 50 as described below. 
[0069] In the preferred embodiment, the apparatus 
further consists of an image processor 140, which proc- 
esses the raw image data. For example, the image proc- 
essor 1 40 may convert the raw image data into a view- 
able image format, such as Joint Photographic Experts 
Group (JPEG). The apparatus may further include a 
main memory 1 50, which stores viewable images after 
processing. 

[0070] Since the linear image combiner 1 20 operates 
on the raw data taken from the sensor and stored in the 
buffer memory 130, nonlinearities are not introduced 
during the image combination process. Additionally, the 
linear image combiner 120 has significant flexibility for 
combining the images. For example, the current raw im- 
age data and image data from the image buffer may be 
summed, averaged, or combined in other linear propor- 
tions. If desired, rather than adding, the pixels may be 
averaged together. Images previously stored in buffer 
memory 130 may also be added. Controlled non-linear 
effects may later be added, if desired. This flexibility is 
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useful when the reason for forming a multiple image is 
something otherthan for providing correct exposure lev- 
els. For example, when it is desired to introduce special 
effects. 

[0071] Reference is now made to Figure 2, which is a 
simplified block diagram of a further preferred embodi- 
ment of an image capture apparatus 200, such as a dig- 
ital camera. Image capture apparatus 200 consists of 
sensor 21 0 r buffer memory 230, and linear image com- 
biner 220, as discussed above. In the preferred embod- 
iment image capture apparatus 200 further consists of, 
separately or in combination, converter 260, controller 
270, display 280, and pointer 290. 
[0072] In the preferred embodiment, converter 260 
converts sensor data to a format required by buffer 
memory 230 and/or combiner 220. Converter 260 may 
function as an analog to digital converter, converting an- 
alog sensor data to digital data. In the preferred embod- 
iment, linear image combiner 220 combines the digi- 
tized current and stored images, after data converter 
260 has performed analog to digital conversion of the 
sensor data. 

[0073] In the preferred embodiment, image capture 
apparatus 200 contains controller 270, which controls 
various apparatus functions. Camera functions, which 
may be controlled by controller 270 and are relevant to 
the present invention, include: activating and deactivat- 
ing multiple exposure mode, adjusting camera settings, 
and taking photographs. Additionally, the controller 270 
may select segments of the multiple exposure image, 
and analyze them to determine whether the required ex- 
posure criterion has been met. Several examples of ex- 
posure criteria are described below. If an additional im- 
age is required, the controller 270 may calculate the re- 
quired exposure, set the camera to the required expo- 
sure, and take another image, which is then added to 
the previous image by the linear image combiner 220. 
The controller 270 may control the camera functions 
based on information received from linear image com- 
biner 220, in order to obtain the desired multiple expo- 
sure image. 

[0074] Linear image combiner 220 may be controlled 
by a user, or automatically by controller 270. Various ex- 
posure criteria may be used by controller 270 as the ex- 
posure criterion. For example, additional images may 
be added to the multiple exposure image until the pixel 
intensity of the combined image or a section of the com- 
bined image reaches a specified exposure threshold. 
The specified segment may be required to have the 
highest detailed brightness, or the average of a shadow 
area in the image may be required to attain a minimum 
predetermined value. If the multiple exposure image is 
below correct exposure and an additional image will sur- 
pass correct exposure, the user or controller may take 
another image or the controller may automatically shoot 
an additional image. Linear image combiner 220 may 
then add a linear fraction of the image to the multiple 
exposure image, so that the resulting image reaches the 



exact required exposure. 

[0075] In the preferred embodiment, image capture 
apparatus 200 also provides a display 280. The display 
280 provides a user with a view of one or more selected 

5 images and/or a histogram providing information about 
the selected images. In the preferred embodiment the 
histogram displays the relative distribution of light inten- 
sity (i.e. the number of pixels at varying intensity levels) 
and/or the exposure status. The histogram indicates the 

10 intensity levels at which large numbers of pixels are 
clustered. The indicated intensity levels can be used to 
select an exposure threshold. The user can use the dis- 
play to display the current image, the multiple exposure 
image, or an image from the memory. In the preferred 

is embodiment, image capture apparatus 200 further con- 
tains a pointer 290, for user interaction with the camera. 
The user may use the pointer to select one or more seg- 
ments of the image, or to select histogram data. 
[0076] The user may use the display to control the 

20 combination of images which form the multiple exposure 
image. For example, the user can select a section of the 
combined image which should have the highest detailed 
brightness. Image capture apparatus 200 then contin- 
ues to take images automatically, comparing the pixel 

25 average in the area pointed at by the user to an overex- 
posure threshold after each image is taken. The camera 
stops taking pictures as soon as the threshold is exceed- 
ed. In an alternate preferred embodiment, the controller 
270 automatically selects the area of the image where 

30 correct exposure is to be measured, for example by se- 
lecting the brightest pixel in the image or some area of 
pixels that are slightly below the brightest value. This 
may be achieved by using the average brightness over 
an area of the image. As an alternative way of automatic 

35 selection, instead of looking at actual pixel brightness 
levels, the threshold may be compared to the intensity 
level of a selected proportion of pixels within the image 
(or selected segment of the image), for example the 
brightest ten percent of the image pixels. 

40 [0077] In state-of-the-art digital cameras the camera 
may comprise a camera body connected to (or commu- 
nicating with) a digital camera back. In the preferred em- 
bodiment, the camera front contains the optical ele- 
ments such as the camera lens, whereas the camera 

45 back contains the sensor, camera controls for setting the 
camera operating modes and for influencing the com- 
putation processes, the camera controller for perform- 
ing the necessary computations for creating multiple ex- 
posure images, and the camera display for displaying 

50 images or image histograms. Alternately, the camera 
may be connected to, and controlled by, a computer 
which performs the necessary display and/or analysis 
functions. 

[0078] Reference is now made to Figure 3, which is a 
55 simplified flowchart of a preferred embodiment of a 
method for creating multiple exposure images by com- 
bining photographed images within camera memory. In 
step 300, a picture is taken to obtain current image data. 



6 



11 



EP1 318 668 A2 



12 



The current image data is stored in memory in step 31 0. 
Images may be converted to digital format prior to stor- 
age in memory. One or more multiple exposure iteration 
steps are next performed, to combine the stored image 
with additional images. 

[0079] A multiple exposure iteration step consists of 
steps 320-340, and is performed as follows. In step 320 
new image data is obtained by taking another image, 
and updating the current image data. Stored image data 
is combined linearly with the current new image data in 
step 330, thereby forming a multiple exposure image. 
The stored image data may consist of a previously gen- 
erated multiple exposure image (for example, the mul- 
tiple exposure image generated during the previous it- 
eration), or a previously photographed single image. 
The resultant multiple exposure image is stored in mem- 
ory in step 340. 

[0080] In the preferred embodiment the multiple ex- 
posure iteration step further contains step 350, in which 
the multiple exposure image is examined to determine 
if the exposure criteria have been attained. Various ex- 
posure criteria may be selected, as discussed above. If 
the multiple exposure image is correctly exposed, the 
multiple exposure process is ended. If the multiple ex- 
posure image is not correctly exposed, another multiple 
exposure iteration step is performed. Adding step 350 
to the multiple exposure iteration step converts the mul- 
tiple exposure creation method into an iterative proce- 
dure. 

[0081] The preferred embodiment contains the further 
step of selecting a manner of combining the image data. 
The multiple exposure image is generated as a linear 
combination of the current image data and previously 
generated image data, for example by summing or av- 
eraging the multiple exposure image of the previous it- 
eration with the current image data, as described above. 
[0082] In the preferred embodiment camera settings 
may be adjusted, either manually or automatically, prior 
to taking an image. In a further preferred embodiment, 
a selected image or information about an image is dis- 
played. The user may use the display to check the mul- 
tiple exposure image, and to select sections of the im- 
age. 

[0083] The abovedescribed embodiments may be 
combined within a digital camera to permit flexible cam- 
era modes of operation. Two examples of operating 
modes for creating multiple exposure images are de- 
scribed below. Figures 4 and 5 are flowcharts, for the 
manual and automatic exposure modes respectively, il- 
lustrating user and camera actions during each step. 

Manual exposure mode 

[0084] 

1) The user selects the option for manual multiple 
exposures. The user sets the camera front to a de- 
sired aperture and speed, and sets the flash equip- 



ment, if present, to the desired settings. If the image 
is to be the same in each exposure, the user en- 
sures that the camera and back will be absolutely 
stable during a sequence of images. 

5 2) The user presses the Shoot button on the camera 
or the camera back. A single regular exposure is 
taken. The raw image resulting from this exposure 
is stored in an image buffer in a random access 
memory in the back. 

io 3) The image is rendered to a viewable image that 
is displayed on the back or on an attached monitor. 
The histogram of light levels in the raw image is also 
computed and displayed. 

4) The user observes the image, the histogram, or 
is both, and judges whether it is necessary to add an 

additional exposure to the current image. The user 
may change a setting affecting the exposure, such 
as the speed, aperture, flash strength, or any other 
camera or lighting control. The histogram display 
20 may also indicate the current state of exposure of 
the image. 

5) The user presses Shoot again. Whereas in the 
single exposure case the image in the image buffer 
is generally flushed for rendering and replaced by 

25 a new image, in the multiple exposure case the im- 
age is not flushed from the buffer. Instead, each pix- 
el of the incoming new image is added to the exist- 
ing raw value at that pixel location in the image buff- 
er, thus creating a multiple exposure image in the 

30 buffer. 

6) The user continues with steps 3, 4, and 5 until 
satisfied that a correctly exposed image has been 
attained. The user then deselects the option for mul- 
tiple manual exposures. The cumulative image is 

35 taken from the image buffer, rendered or otherwise 
processed, and saved in camera memory in a sim- 
ilar manner to a single exposure image. 

Automatic exposure mode 

40 

[0085] 

1 ) The user selects the option for automatic multiple 
exposures. The user sets the camera body to a de- 

45 sired aperture and speed, and sets the flash equip- 
ment, if present, to the desired settings. If the image 
is to be the same in each exposure, the user en- 
sures that the camera and back will be absolutely 
stable during a sequence of images. 

so 2) The user presses the Shoot button on the cam- 
era or the camera back. A single regular exposure 
is taken. The raw image resulting from this expo- 
sure is stored in an image buffer in a random access 
memory in the back. 

55 3) The image is rendered to a viewable image that 
is displayed on the back or on an attached monitor. 
The histogram of light levels in the raw image is also 
computed and displayed. The back displays a mes- 
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sage requiring the user to interact with it. 

4) The user observes the image and points at an 
area on the display image that should obtain the 
highest detailed brightness, i.e. all brighter pixels 
will be overexposed. The back resumes function- 
ing. 

5) The back continues to shoot a sequence of im- 
ages, and to add each image to the cumulative im- 
age as described in above step 5 of the manual ex- 
posure mode. After each image, the image combin- 
er examines the pixel average in the area pointed 
to by the user. If the average is less than a prede- 
termined overexposure threshold, and addition of 
an identical subsequent image will not pass the 
threshold, an additional image is shot and added to 
the cumulative image in the buffer. If the overexpo- 
sure threshold has been reached, the shooting loop 
is terminated, and the cumulative multiple exposure 
image is transferred from the image buffer to cam- 
era memory. 

[0086] Multiple exposure photography is a powerful 
tool for both professional and amateur photographers. 
The digital camera described above provides photogra- 
phers with a simple and accurate way of forming multiple 
exposure images. The images may be formed directly 
within the camera, with no need for further processing. 
Undesired non linearities are not introduced, and the im- 
age combination process may be controlled simply and 
precisely. 

[0087] It is appreciated that certain features of the in- 
vention, which are, for clarity, described in the context 
of separate embodiments, may also be provided in com- 
bination in a single embodiment. Conversely, various 
features of the invention which are, for brevity, described 
in the context of a single embodiment, may also be pro- 
vided separately or in any suitable subcombination. 
[0088] Unless otherwise defined, all technical and sci- 
entific terms used herein have the same meanings as 
are commonly understood by one of ordinary skill in the 
art to which this invention belongs. Although methods 
similar or equivalent to those described herein can be 
used in the practice or testing of the present invention, 
suitable methods are described herein. 
[0089] All publications, patent applications, patents, 
and other references mentioned herein are incorporated 
by reference in their entirety. In case of conflict, the pat- 
ent specification, including definitions, will prevail. In ad- 
dition, the materials, methods, and examples are illus- 
trative only and not intended to be limiting. 
[0090] It will be appreciated by persons skilled in the 
art that the present invention is not limited to what has 
been particularly shown and described hereinabove. 
Rather the scope of the present invention is defined by 
the appended claims and includes both combinations 
and subcombinations of the various features described 
hereinabove as well as variations and modifications 
thereof which would occur to persons skilled in the art 



upon reading the foregoing description. 



Claims 

5 

1 . Image capture apparatus for the formation of a mul- 
tiple exposure image, comprising: 

an image capture sensor for capturing images, 
10 and 

a linear image combiner, associated with said 
image capture sensor, for linearly combining 
raw image capture data from said sensor, 
thereby to form said multiple exposure image. 

15 

2. Image capture apparatus according to claim 1 , fur- 
ther comprising a buffer memory at said sensor out- 
put for storing said raw image data, said buffer 
memory being operatively associated with said lin- 

20 ear image combiner to allow said image combiner 
access to said raw image data. 

3. Image capture apparatus according to claim 2, 
wherein said linear image combiner is further oper- 

25 able to store said multiple exposure image in said 
buffer memory. 

4. Image capture apparatus according to claim 1 , 2, or 
3, being a digital camera. 

30 

5. Image capture apparatus according to any preced- 
ing claim, wherein said linear image combiner is op- 
erable to form said multiple exposure image as a 
linear combination of stored raw image data and a 

35 current raw image data capture. 

6. Image capture apparatus according to claim 5, 
wherein said linear combination is one of a group 
comprising: a linear fraction of a sum of said stored 

40 raw image data and said current raw image data 
capture, an average of said stored raw image data 
and said current raw image data capture, and a sum 
of said stored raw image data and a linear fraction 
of said current raw image data capture. 

45 

7. Image capture apparatus according to any preced- 
ing claim, wherein said linear image combiner is 
controllable by a user. 

50 8. Image capture apparatus according to any preced- 
ing claim, wherein said linear image combiner is op- 
erable to combine said raw image data automatical- 
ly, so as to satisfy a predetermined exposure crite- 
rion for said multiple exposure image. 

55 

9. Image capture apparatus according to claim 8, 
wherein said criterion comprises having a pixel in- 
tensity level within a segment of said multiple expo- 
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sure image exceed a predetermined threshold. 

10. Image capture apparatus according to claim 2, 
wherein said linear image combiner is operable to 
form said multiple exposure image from more than 
one image stored in said buffer memory. 

11 . Image capture apparatus according to any preced- 
ing claim, further comprising a controller, for con- 
trolling apparatus functions. 

12. Image capture apparatus according to claim 11, 
wherein said controller is operable to control said 
functions in accordance with information obtained 
from said image combiner. 

13. Image capture apparatus according to claim 11, 
wherein one of said functions comprises activating 
and deactivating a multiple exposure mode. 

14. Image capture apparatus according to claim 11, 
wherein one of said functions comprises selecting 
a segment of an image. 

15. Image capture apparatus according to claim 11, 
wherein one of said functions comprises selecting 
and deselecting an automatic multiple exposure 
mode. 

16. Image capture apparatus according to claim 11, 
wherein one of said functions comprises capturing 
an image. 

17. Image capture apparatus according to claim 11, 
wherein said apparatus is a digital camera and one 
of said functions comprises controlling camera set- 
tings. 

18. Image capture apparatus according to claim 17, 
wherein said settings comprise camera aperture, 
camera speed, and flash operation. 

19. Image capture apparatus according to any preced- 
ing claim, further comprising a converter, for con- 
verting current raw image data into a digitized im- 
age. 

20. Image capture apparatus according to claim 2, 
wherein said buffer memory comprises a random 
access memory (RAM). 

21 . Image capture apparatus according to any preced- 
ing claim, further comprising a display. 

22. Image capture apparatus according to claim 21, 
wherein said display is operable to display a select- 
ed image. 



23. Image capture apparatus according to claim 22, 
wherein said selected image is one of a group com- 
prising: a current image, said multiple exposure im- 
age, and an image stored in said buffer memory. 

5 

24. Image capture apparatus according to claim 21 , 
wherein said display is operable to display a histo- 
gram for a selected image. 

io 25. Image capture apparatus according to claim 24, 
wherein said histogram indicates the relative distri- 
bution of light intensities for said selected image. 

26. Image capture apparatus according to claim 24, 
15 wherein said histogram indicates a current state of 

exposure of said selected image. 

27. Image capture apparatus according to claim 24, 
wherein said selected image is one of a group com- 

20 prising: a current image, said multiple exposure im- 
age, and an image from buffer memory. 

28. Image capture apparatus according to claim 21 , fur- 
ther comprising a pointer, for selecting a segment 

25 of a displayed image. 

29. Image capture apparatus according to any preced- 
ing claim, wherein said sensor comprises a charge- 
coupled device (CCD) sensor. 

30 

30. Image capture apparatus according to any preced- 
ing claim, wherein said sensor comprises a comple- 
mentary metal oxide semiconductor (CMOS) sen- 
sor. 

35 

31 . Image capture apparatus according to any preced- 
ing claim, further comprising an image processor for 
processing raw image data. 

40 32. Image capture apparatus according to claim 31, 
wherein said processing comprises rendering said 
raw image data into a viewable image. 

33. Image capture apparatus according to claim 31 , fur- 
45 ther comprising a main memory for storing said 

processed image data. 

34. A method for creating multiple exposure images by 
linearly combining raw capture data from a sensor, 

so comprising: 

obtaining current raw image data; 

storing said current raw image data in memory; 

and 

55 performing a multiple exposure iteration step 

comprising: 

obtaining new current raw image data; 
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linearly combining stored raw image data 
with said current raw image data to form a 
multiple exposure image; and 
storing said multiple exposure image in 
said memory. 

35. A method forcreating multiple exposure images ac- 
cording to claim 34, wherein said multiple exposure 
iteration step further comprises the steps of: 

determining if said multiple exposure image is 
correctly exposed; 

if said multiple exposure image is correctly ex- 
posed, discontinuing said multiple exposure 
process; and 

if said multiple exposure image is not correctly 
exposed, performing a further multiple expo- 
sure iteration step. 

36. A method forcreating multiple exposure images ac- 
cording to claim 34 or claim 35, wherein said stored 
raw image data comprises more than one image 
stored in said memory. 

37. A method forcreating multiple exposure images ac- 
cording to claim 34, 35 or 36, wherein said method 
is performed by a digital camera. 

38. A method forcreating multiple exposure images ac- 
cording to claim 37, further comprising adjusting 
camera settings. 

39. A method forcreating multiple exposure images ac- 
cording to claim 37, wherein said multiple exposure 
iteration step further corhprises adjusting camera 
settings. 

40. A method for creating multiple exposure images ac- 
cording to claim 38, further comprising adjusting 
said camera settings automatically. 

41 . A method forcreating multiple exposure images ac- 
cording to claim 38, wherein said multiple exposure 
iteration step further comprises adjusting said cam- 
era settings automatically. 

42. A method forcreating multiple exposure images ac- 
cording to any one of claims 34 to 41 , wherein said 
multiple exposure iteration step further comprises 
displaying a selected image. 

43. A method forcreating multiple exposure images ac- 
cording to claim 42, wherein said selected image is 
one of a group comprising: a current image, said 
multiple exposure image, and an image stored in 
said memory. 

44. A method forcreating multiple exposure images ac- 



cording to claim 42, wherein said multiple exposure 
iteration step further comprises selecting at least 
one segment of said displayed image. 

5 45. A method forcreating multiple exposure images ac- 
cording to claim 44, further comprising analyzing 
said selected segments to determine image expo- 
sure. 

io 46. A method for creating multiple exposure images ac- 
cording to any one of claims 34 to 45, wherein said 
multiple exposure iteration step further comprises 
displaying a histogram of a selected image. 

15 47. a method for creating multiple exposure images ac- 
cording to claim 46, wherein said selected image is 
one of a group comprising: a current image, said 
multiple exposure image, and an image stored in 
said memory. 

20 

48. A method forcreating multiple exposure images ac- 
cording to claim 46, further comprising analyzing 
said histogram to determine image exposure. 

25 49. A method forcreating multiple exposure images ac- 
cording to claim 46, wherein said histogram indi- 
cates the relative distribution of light intensities for 
said selected image. 

30 50. A method for creating multiple exposure images ac- 
cording to claim 46, wherein said histogram indi- 
cates a current state of exposure of said selected 
image. 

35 51 . A method for creating multiple exposure images ac- 
cording to any one of claims 34 to 50, further com- 
prising selecting a manner of combining said imag- 
es. 

40 52. A method for creating multiple exposure images ac- 
cording to claim 51 , wherein said images are com- 
bined automatically according to said selected 
manner. 

45 53. a method forcreating multiple exposure images ac- 
cording to any one of claims 34 to 52, further com- 
prising selecting a criterion for determining rf said 
multiple exposure image is correctly exposed. 

50 54. a method for creating multiple exposure images ac- 
cording to claim 53, wherein said criterion compris- 
es having a pixel intensity level within a segment of 
said multiple exposure image exceed a predeter- 
mined threshold. 

55 

55. A method for creating multiple exposure images ac- 
cording to claim any one of claims 34 to 54, wherein 
said memory comprises a RAM. 
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56. A method for creating multiple exposure images ac- 
cording to any one of claims 34 to 55, further com- 
prising digitizing said current raw image data. 

57. Image capture apparatus comprising a sensor for 
capturing raw image data, an image buffer for stor- 
ing said captured raw image data, an image proc- 
essor for processing said captured data into a dis- 
playable image file and a memory for storing said 
image file, the apparatus further comprising an im- 
age combiner associated with said image buffer for 
performing linear combinations between different 
captures of said raw image data, therefrom to form 
multiple exposure images. 

58. The image capture apparatus of claim 57, wherein 
said image combiner is operable to continue to ac- 
cumulate combinations using succeeding image 
captures until a predetermined light level is reached 
within said combination. 
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